A growing diversity of client devices are gaining access to networked servers that distribute rich multimedia content. However, the capabilities of the devices to access, process and display the content varies widely. While color workstations, which have high bandwidth network connections, can readily access and display large colorful images, many hand-held computers (HHCs), personal digital assistants (PDAs), screen phones, and smart phones can only display small images and cannot handle video. Television-based web browsers are constrained by the low-resolution interlaced display of television screens. Personal computers having color monitors often achieve low data rates along dial-up network connections, thereby inhibiting access to rich content. Given the variety of client devices, it is difficult for content publishers to anticipate and accommodate the wide spectrum of client capabilities.
Options for content adaptation include developing multiple versions of multimedia content, each suitable for a different class of client devices. Manually generating multiple versions works well if the devices can be easily aggregated into a small number of classes. Alternatively, methods can be developed that automatically generate the multiple versions of the content, such as creating a full-resolution version of the content which can be processed to generate lower resolution versions. The latter approach can be extended to allow content servers to automatically generate the appropriate version of the content at the time of request. The server can manipulate, or transcode, the existing full-resolution content, on-the-fly, to adapt it to constraints in delivery and constraints in display, processing, and storage at the client devices.
The transcoding mechanism can be deployed in a number of ways in a networked system, including deployment at a server or at the client. Alternatively, the transcoding system can be deployed at a proxy which retrieves the content from the content server, manipulates it on-the-fly, and forwards the results to the client device, as demonstrated by J. R. Smith, R. Mohan, and C.-S. Li, in an article entitled “Transcoding Internet content for heterogeneous client devices”, published in Proc. IEEE Inter. Symp. On. Circuits and Syst. (ISCAS), June, 1998. A proxy system can optionally cache different versions of the content to speed up the transcoded content delivery. Proxy-based transcoding systems have been developed for adapting images to client devices. Fox, et al., developed a system for compressing images that pass through the network proxy device, as detailed in “Adapting to network and client variability via on-demand dynamic distillation”. published in ASPLOS-VII, Cambridge, Mass., October, 1996.
Other systems that compress the images using a proxy implementation to speed-up image download time (see: e.g., Intel Quick Web. Http://www.intel.com/quickweb and Spyglass Prism. Http://www.spyglass.com/products/prism.
There are many ways in which a transcoder can adapt content to the client device, such as by data compression, summarization and media conversion. Benefits can be realized by selecting the transcoding operations on the basis of the network conditions, publisher preferences, user preferences and the client device capabilities. Furthermore, additional benefits could be gained by selecting the transcoding operations on the basis of an analysis of the content as demonstrated by J. R. Smith, R. Mohan and C.-S. Li in an article entitled “Content-based transcoding of images in the Internet,” published in Proc. of IEEE Inter. Conf. On Image Processing (ICIP-98), Chicago, Ill., October 1998, and in an article entitled “Multimedia content customization for universal access,” published in Proc. of SPIE East-Multimedia Storage and Archiving Systems III, Boston, Mass., November 1998.
There are many dimensions by which the content could be analyzed in order to select the transcoding operations. For example, the content analysis can ideally examine any of the following: the visual, audio, or textual characteristics of the content, such as the color information in images, the motion or scene information in video, spectral information in audio, or the occurrence of words in text passages; the purpose of the content in the larger context of a multimedia document, such as by identifying titles, headings, paragraphs, abstracts, advertisements, and inter-document links; or the importance or relevance of the content in the document or to the user, such as by identifying paragraphs related to search terms, images related to query images, or multi-media objects related to specific semantic classes.
On the basis of the content analysis, the transcoding system could then select different transcoding operations for different classes of content. For example, the transcoding system could selectively compress color and black-and-white images differently; could detect audio passages that have characteristics of speech, then convert the speech to text; could selectively remove advertisement graphics and leave other images; or could selectively and lossily compress objects within a multimedia document based on their relevance to a semantic topic or to search terms in order to conserve bandwidth. By coupling the content analysis with transcoding, the content could be better adapted to constraints in delivery, display, processing and storage.
It is, therefore, an objective of the present invention to provide a system and method for analyzing multimedia content prior to transcoding same for delivery.
It is another objective of the invention to selectively transcode multimedia content based on content analysis.